3 x 3 matrix for unitary optical systems.

We introduce a 3 x 3 matrix for the study of unitary optical systems. This 3 x 3 matrix is a submatrix of the 4 x 4 Mueller matrix. The elements of this 3 x 3 matrix are real, and thus complex-number calculations can be avoided. The 3 x 3 matrix is useful for illustrating the polarization state of an optical system. One can also use it to derive the conditions for linear and circular polarization output for a general optical system. New characterization methods for unitary optical systems are introduced. It is shown that the trajectory of the Stokes vector on a Poincaré sphere is either a circle or an ellipse as the optical system or input polarizer is rotated. One can use this characteristic circle or ellipse to measure the equivalent optical retardation and rotation of any lossless optical system.